Learning coach for machine learning system

ABSTRACT

A machine learning system includes a coach machine learning system that uses machine learning to help a student machine learning system learn its system. By monitoring the student learning system, the coach machine learning system can learn (through machine learning techniques) “hyperparameters” for the student learning system that control the machine learning process for the student learning system. The machine learning coach could also determine structural modifications for the student learning system architecture. The learning coach can also control data flow to the student learning system.

PRIORITY CLAIM

The present application is a continuation under 35 U.S.C. § 120 of U.S.patent application Ser. No. 16/334,204, filed Mar. 18, 2019, whichapplication is a National Stage of International Application No.PCT/US2017/052037, filed Sep. 18, 2017, which claims priority to thefollowing U.S. provisional patent applications: (1) Ser. No. 62/400,853,filed Sep. 28, 2016, entitled “Deep Learning With Socratic Coaches”; and(2) Ser. No. 62/476,280, filed Mar. 24, 2017, entitled “Learning Coachfor Machine Learning System.” These two provisional applications areincorporated herein by reference in their entirety.

CROSS-REFERENCE TO RELATED APPLICATIONS

U.S. application Ser. No. 16/891,866, filed Jun. 3, 2020, now U.S. Pat.No. 10,839,294, issued Nov. 17, 2020, also is a continuation of U.S.patent application Ser. No. 16/334,204.

BACKGROUND

Machine learning is a process implemented by computers to self-learnalgorithms that can make predictions on data through building modelsfrom sample data inputs, without being explicitly programmed to do so.There are many types of machine learning system types, such asartificial neural networks (ANNs), decision trees, support vectormachines (SVMs), and others. These systems first have to be trained onsome of the sample inputs before making meaningful predictions with newdata. For example, an ANN typically consists of multiple layers ofneurons. Each neuron is connected with many others, and links can beenforcing or inhibitory in their effect on the activation state ofconnected neurons. Each individual neural unit may have a summationfunction which combines the values of all its inputs together. There maybe a threshold function or limiting function on each connection and onthe neuron itself, such that the signal must surpass the limit beforepropagating to other neurons. The weight for each respective input to anode can be trained by back propagation of the partial derivative of anerror cost function, with the estimates being accumulated over thetraining data samples. A large, complex ANN can have millions ofconnections between nodes, and the weight for each connection has to belearned.

SUMMARY

In one general aspect, the present invention is directed to a computersystem that comprises one or more “student” machine learning systemsalong with at least one “coach” machine learning system. The coachmachine learning system itself uses machine learning to help the studentmachine learning system(s). For example, by monitoring a student machinelearning system, the coach machine learning system can learn (throughmachine learning techniques) “hyperparameters” for the student machinelearning system that control the machine learning process for thestudent learning system. For example, in the case where the studentmachine learning system uses a deep neural network (DNN), the learnedhyperparameters can include the minibatch size M, the learning rate η,the regularization parameter λ, and/or the momentum parameter μ. Also,one set of learned hyperparameters could be used to determine all of theweights of the student machine learning system's network, or customizedlearned hypermeters can be used for different weights in the network.For example, each weight (or other trainable parameter) of the studentlearning system could have its own set of customized learnedhyperparameters that are learned by the learning system coach.

Additionally or in lieu of learning the hyperparameters, the machinelearning coach could determine structural modifications for the studentlearning system architecture. For example, where the student learningsystem uses a DNN, the machine learning coach can modify the structureof the DNN, such as by adding or deleting layers and/or by adding ordeleting nodes in layers. Additionally, the student learning systemmight include an ensemble of machine learning system. The learning coachin such a scenario could control the data flow to the various machinelearning systems and/or add members to the ensemble.

The student learning system(s) and machine learning coach preferablyoperate in parallel. That is, the machine learning coach observes thestudent learning system(s) while the student learning system(s) is/arein the learning process and the machine learning coach makes its changesto the student learning system(s) (e.g., hyperparameters, structuralmodifications, etc.) while the student learning system(s) is/are in thelearning process. The learning coach and the student(s) may be the sameor different types of machine learning architectures.

The learning coach can have an objective function distinct from theobjective of the student learning system(s). For example, the studentlearning system may minimize the error rate or some other measure of thecost of the errors, while the learning coach makes structuralmodifications to the student learning system to optimize somecombination of the cost of errors and the cost of performing thecomputation. The learning coach can also make modifications to thestudent learning system, especially additions, to improve itscapabilities while guaranteeing that there will be no degradation inperformance.

These and other benefits of the present invention are apparent from thedescription herein.

FIGURES

Various embodiments of the present invention are described herein by wayof example in connection with the following figures, wherein:

FIGS. 1, 14 and 15 are block diagrams of systems that include a studentmachine learning system and a learning coach according to variousembodiments of the present invention;

FIG. 2 is a block diagram of one process by which the learning coachcontrols the training of a student learning system according to variousembodiments of the present invention;

FIG. 3 is an illustration of a process according to various embodimentsof the present invention by which the learning coach collectsobservations of the student learning system as the student learningsystem is being trained;

FIG. 4 is a block diagram of large learning system with data flowcontrolled by learning coaches according to various embodiments of thepresent invention;

FIG. 5 is a block diagram of the system of FIG. 4 where some of thelearning modules are stored in high-speed memory or cache and others arestored in secondary storage according to various embodiments of thepresent invention;

FIG. 6 illustrates a process for transferring explicit knowledge to amachine learning network according to various embodiments of the presentinvention;

FIGS. 7 and 8 illustrate mechanisms for adding nodes to a learningnetwork according to various embodiments of the present invention;

FIG. 9 is a flow chart that illustrates a process for creating explicitknowledge according to various embodiments of the present invention;

FIG. 10 is a block diagram that illustrates a system for making implicitknowledge communicable according to various embodiments of the presentinvention;

FIG. 11 is a block diagram of a large machine learning system that shareknowledge among machine learning modules of the system according tovarious embodiments of the present invention;

FIG. 12 is a block diagram of an ensemble machine learning system wherethe learning coach controls the distribution of data among ensemblemembers according to various embodiments of the present invention; and

FIG. 13 is a block diagram of an ensemble machine learning system wherea preliminary classifier controls the distribution of data amongensemble members according to various embodiments of the presentinvention.

DESCRIPTION

FIG. 1 is a block diagram of a computer system 10 according to a basicembodiment of the present invention. FIG. 1 shows a machine learningsystem 11 that is being trained to recognize patterns in the datasupplied from a data source 21. Since the machine learning system 11 isin the training or learning process, it is also referred to herein as a“student.” The system 10 further includes a machine learning system 41that controls some aspects of the learning process and aids the studentmachine learning system 11 in its process of learning the patterns.Because it helps that learning process, the machine learning system 41is called a “learning coach.”

In various embodiments, the learning coach 41 is itself a machinelearning system. However, the learning coach 41 does not try torecognize the patterns that the student learning system 11 is learningto recognize. The primary data processed by the learning coach 41 areobservations 31 of the internal state of the student machine learningsystem 11 and data that student system 11 computes during its trainingprocess. In addition, among the control mechanisms used by the learningcoach 41 is the selection of training data for the student 11. Thus, thelearning coach 41 may also receive the training data 34 as input, butnot for the purpose of recognizing patterns in it, as described herein.The learning coach 41 preferably also has other means of controlling andaiding the learning process of the student 11, including controllinghyperparameters 32 of the student 11 and changing the internal structure33 of the student machine learning system 11, as described herein.

For purpose of discussion, focus first on the controls called“hyperparameters.” Generally, any large complex machine learning system,such as the student learning system 11, has a large number of parametersfor which good values need to be found to try to minimize some measureof the cost of errors in the pattern recognition process. These are thetrainable parameters. Finding these values is the essence of the processcalled “learning” in a machine learning system. There may be millions oreven billions of these parameters, so the learning process may be verytime-consuming and expensive. To help the learning process work better,there are usually a few control parameters that tune the learningprocess to work better on the kind of data for a particular task. Thesecontrol parameters are not learned by the machine learning system 11,but usually are specified by the system designer. To distinguish themfrom the parameters being learned by system 11, these control parametersare referred to as “hyperparameters” herein; they are parameters thatcontrol the learning process of the machine learning system.

In embodiments of the present invention, some or all of thesehyperparameters are not controlled by the system designer, but ratherare controlled be the learning coach 41.

To illustrate the use of hyperparameters through an illustrativeexample, let the student system 11 be a large, deep feed-forward neuralnetwork (a DNN), a type of machine learning system that is well known tothose skilled in the art of machine learning. A deep neural network is anetwork that has many layers of nodes representing artificial models ofneurons with connections or arcs in the network representing thestrength of the connection between the neurons represented by the nodes.Each arc is associated with a numerical value that represents thestrength or “weight” of its connection. These weights comprise theparameters to be learned in the training of student system 11. In alarge DNN, there may be millions of nodes with billions of connectionsand, therefore, billions of these connection weights that need to belearned. The individual nodes and arcs can use a greatly simplified,artificial model of a real neuron. The complexity of the learningprocess for deep neural networks arises from the large number of nodesand arcs and the complexity of their interactions.

For purpose of illustration, the use of the hyperparameters for trainingdeep feed forward neural networks will be described. This example is byway of illustration and is not intended as limiting the scope of theinvention, as the invention can be applied to any complex machinelearning system, including support vector machines (SVMs), randomforests, CART analysis, hidden stochastic process models, dynamicBayesian networks, Boltzmann machines, ensembles of any of these andeven heterogeneous ensembles. The machine learning system may be complexin any one or more of several ways. It may have a large, complexinternal structure; its learning may be controlled by hyperparameters;or its training may be done by a computationally intensive process on alarge amount of training data. The illustrative training method is aniterative process of stochastic gradient descent on a log-likelihooderror cost function, with a softmax classification for the output nodes,the gradients for each training example being estimated by backpropagation of the partial derivative of the error cost function, theestimates being accumulated over minibatches, with one update of theparameters for each minibatch. In the illustrative method, the trainingprocess is enhanced by Nesterov's momentum, and smoothed by L2regularization. Other well-known training methods could be used in asimilar way.

The pseudo-code for this well-known training process is as follows:

Initialize each weight; Do until a stopping criterion is reached: {  1.Input a set (minibatch) of training examples  2. For each trainingexample m, set a_(0,i)(m) and perform the following   steps:    a.Feedforward (softmax output): For each l = 1, 2, ..., L − 1     compute${{z_{l,j}(m)} = {\sum\limits_{i = 0}^{n_{l}}{w_{{l - 1},i,j}{a_{{l - 1},i}(m)}}}},{{a_{l,j}(m)} = {\sigma\left( {z_{l,j}(m)} \right)}},{a_{L,k} = {e^{z_{L,k}}/{\sum\limits_{j}e^{z_{L,j}}}}}$   b. Output error gradient δ_(L,j)(m):     δ_(L,j)(m) = −(y_(j)(m) −a_(L,j)(m))/n_(L)    c. Backpropagate error gradient: For each 1 =L-1,L-2,. . . ,2, 1     compute    ${\delta_{{l - 1},i}(m)} = {{a_{{l - 1},i}(m)}\left( {1 - {a_{{l - 1},i}(m)}} \right){\sum\limits_{j = 1}^{n_{l}}{w_{{l - 1},i,j}{\delta_{l,j}(m)}}}}$  3. Gradient descent: For each 1 = L-1, L-2, . . . ,2, 1 update theweights      = v_(l, i, j)$\left. v_{l,i,j}\leftarrow{{\mu v_{l,i,j}} - {\eta{\sum\limits_{m = 1}^{M}{{a_{l,i}(m)}{{\delta_{{l + 1},j}(m)}/M}}}}} \right.$$\left. w_{l,i,j}\leftarrow{{w_{l,i,j}\left( {1 - \frac{\eta\lambda}{n}} \right)} - {\mu\;} + {\left( {1 + \mu} \right)v_{l,i,j}}} \right.$Each weight w_(l,i,j) may be initialized as a random number drawn fromthe normal distribution

${N\left( {{\mu = 0},{\sigma = \frac{1}{\sqrt{n_{j}}}}} \right)},$where w_(i,j) is the connection weight between node i in layer k−1 andnode j in layer k, and n_(j) is the number of connections into noden_(j). There are L layers, counting the output layer. There are Mexamples in the minibatch. For each layer 1<L, there is a dummy node 0that always has the activation 1.0, so the weight w_(l,0,j) acts like abias for node j in layer l+1.

This method of training deep feed forward neural networks is known andsuccessful. However, it is only one example of many variations. Thisinvention applies to any of these variations in a manner like thisillustrative example. In the prior art, M, η, λ, μ, and otherhyperparameters must be determined by experimentation. In embodiments ofthe present invention, learning coach 41 learns to optimize thesehyperparameters from observation of the learning process of studentlearning system 11. Since these hyperparameters have numerical values,many as continuous variables, learning coach 41 may model the learningperformance of student learning system 11 as a regression. Alternately,or in addition, learning coach 41 may directly estimate the optimumvalue of a hyperparameter as a regression function of observations thatlearning coach 41 has made during the current or previous steps of thetraining of student learning system 11. Learning coach 41 may learncategorical hyperparameters as a classification task.

Note that there are several hyperparameters that control the training,including for example M (the size of the minibatch), η (the learningrate), λ (the regularization parameter) and μ (the momentum parameter).Generally, the values of these parameters are set by trial-and-error, avery tedious process since each trial requires a test run of the fulltraining procedure on a large amount of training data and then testingon a separate validation set. The process can be automated in the sensethat an entire n-dimensional grid can be explored automatically, butthat is even more computationally expensive.

Furthermore, in the illustrative example, each of these hyperparametersis a constant applied to all parameters in the network. That is, thesame value is used for every weight in the network, which may havemillions of nodes and billions of connections. Some improvement can beachieved by using any of several schemes that allow the learning ratehyperparameter, to be different for each weight. A technique calledRMSprop is an illustrative example of one of these schemes.

In RMSprop, there is a different learning rate for each weight,depending on the size of its past partial derivatives. The justificationis that, if a weight has had many large partial derivatives in the past,then it no longer needs to have a learning rate as high as otherweights. However, RMSprop only adds one extra degree of control becausethe change depends on a single statistic. This one extra degree ofcontrol hardly begins to approach the problem of customizing thelearning rate to all the different conditions of the weights beingtrained in various parts of the network at the various stages of thelearning process. Of course, it would be impossible for a human toindividually customize the learning schedule for each of up to over abillion parameters.

However, in the illustrative embodiment of the invention, the learningcoach 41 can customize the hyperparameters of the student learningsystem 11. In particular, FIG. 2 illustrates an embodiment of a processfor the learning coach 41 to control various aspects of the studentlearning system 11. As shown in FIG. 2, the learning coach 41 cancontrol hyperparameters of the student learning system 11 at block 32C;modify the structure of the student learning system 11 at block 33C asdescribed herein; and/or control data selection for the student learningsystem 11 at block 34C as described herein. Block 11A in FIG. 2 providesan example learning algorithm for the student learning system 11. Eachhyperparameter η_(l,i,j),λ_(l,i,j),μ_(l,i,j) in the algorithm hassubscripts l,i,j. Even the minibatch size, M, can be customized todifferent value M_(l,i,j) for each connection weight. This customizationof M may be achieved by postponing the update for the weight to somemultiple of the default value of M. Each connection weight can use acustomized value of the multiple. Thus each weight, that is, eachparameter being trained, can have a unique set of values for thehyperparameters.

Each hyperparameter is replaced by N customized hyperparameterscontrolled by learning coach 41, where N is the total number of weights(or other learned parameters). In a large neural network, N may be inthe billions. The pseudo-code now becomes as shown in box 11A of FIG. 2:

Initialize each weight; Do until a stopping criterion is reached: {  3.Input a set (minibatch) of training examples  4. For each trainingexample m, set a_(0,i)(m) and perform the following   steps:    a.Feedforward (softmax output): For each 1 = 1, 2, ..., L − 1     compute${{z_{l,j}(m)} = {\sum\limits_{i = 0}^{n_{l}}{w_{{l - 1},i,j}{a_{{l - 1},i}(m)}}}},{{a_{l,j}(m)} = {\sigma\left( {z_{l,j}(m)} \right)}},{a_{L,k} = {e^{z_{L,k}}/{\sum\limits_{j}e^{z_{L,j}}}}}$   b. Output error gradient δL,j(m):      δ_(L,j)(m) = −(y_(j)(m) −a_(L,j)(m))/n_(L)    c. Backpropagate error gradient: For each 1 =L-1,L-2, . . . ,2, 1     compute    ${\delta_{{l - 1},i}(m)} = {{a_{{l - 1},i}(m)}\left( {1 - {a_{{l - 1},i}(m)}} \right){\sum\limits_{j = 1}^{n_{l}}{w_{{l - 1},i,j}{\delta_{l,j}(m)}}}}$  4. Gradient descent: For each 1 = L-1, L-2, . . . ,2, 1 update theweights    = v_(l, i, j)$\left. v_{l,i,j}\leftarrow{{\mu_{l,i,{jk}}v_{l,i,j}} - {\eta_{l,i,j}{\sum\limits_{m = 1}^{M}{{a_{l,i}(m)}{{\delta_{{l + 1},j}(m)}/M_{l,i,j}}}}}} \right.$$\left. w_{l,i,j}\leftarrow{{w_{l,i,j}\left( {1 - \frac{\eta_{l,i,j}\lambda_{l,i,j}}{n}} \right)} - {\mu_{l,i,j}\;} + {\left( {1 + \mu_{l,i,j}} \right)v_{l,i,j}}} \right.$

Similar customized hyperparameters may be controlled by learning coach41 for other neural network training algorithms and for the trainingalgorithms of other machine learning systems. Although this illustrativeembodiment has been described with respect to a set of hyperparameterscommonly used for training deep neural networks, it should be understoodthat the invention can apply the same process for the learning coach 41to learn to estimate the best values for the hyperparameters for thetraining of any type of student machine learning system 11.

As a machine learning system, the learning coach 41 can learn toestimate the optimum value for each customized hyperparameter based onthe observed values of variables related to a particular connectionweight and its surrounding nodes. For each training example m in eachminibatch k, it can observe the results of the forward activation of thedeep neural network and the backpropagation of the partial derivatives.Note that it can observe these values for each example in the minibatch,not just summary statistics once for each update in the minibatchgradient descent.

As illustrated in blocks 31 and 35 of FIG. 3, each update of eachconnection weight parameter gives the learning coach 41 an example ofthe effect of a set of hyperparameter values on the future learningperformance of student learning system 11. Learning coach 41 models thiseffect as a function of the hyperparameters and of the context, that is,the set of observations surrounding the connection for each example inthe preceding minibatch or multiple preceding minibatches. In otherwords, the learning coach has millions of training examples from whichit can learn to predict the best hyperparameter values for eachcustomized situation.

One aspect of embodiments of the present invention is that the learningcoach 41 learns to recognize patterns in the observations of anindividual weight parameter and patterns in nearby nodes and connectionsin the network. Learning coach 41 learns these patterns from observingthe learning process of many different examples of student learningsystems 11. That is, learning coach 41 learns from prior experience ofcoaching other student learning systems 11, not just from the currentexample.

As will be explained in more detail in reference to other figures, thelearning coach 41 can apply a similar learning process to its ability tomodify the structure of the student learning system 11, as in block 33Cof FIG. 2 (e.g., adding or removing layers from the network in instanceswhere the student learning system 11 uses a network) and its ability tocontrol the selection of training data supplied to the student learningsystem, as in block 34C of FIG. 2.

Another general aspect of the invention according to various embodimentsis that the learning coach 41 can make structural changes in studentlearning system 11, as shown by control line 33 in FIG. 1. For example,if student learning system 11 is a neural network, learning coach 41 canadd or delete connections, nodes, or groups of nodes, even entire layersof nodes. When learning coach 41 makes a decision, such as a structuralchange, or estimates a discrete variable, rather than a continuousvariable or hyperparameter, it can represent its task as aclassification problem rather than as a regression. For example, iflearning coach 41 is a neural network, it can use a softmax function ona set of output nodes rather than linear nodes.

When learning coach 41 makes structural changes to the student learningsystem 11, it affects not just the learning rate but also the minimumcost. It also affects the amount of computation required by studentlearning system 11. One aspect of the present invention according tovarious embodiments is that learning coach 41 can have an objective thatis different from the objective of student learning system 11. Forexample, in evaluating structural changes, learning coach 41 mayoptimize some combination of cost of computation and some measure of theerror rate of a given structure, whereas student learning system 11 justminimizes some measure of the cost of errors for a fixed structure.

In one illustrative embodiment, learning coach 41 trains itself to makedecisions about structural changes in student learning system 11 byrunning experiments. In this illustrative embodiment, learning coach 41implements two versions of student learning system 11, one with thestructural change and one without. Learning coach trains both versionsof student learning system 11 and evaluates their comparativeperformance on an independent evaluation set. It also measures orestimates the comparative computational requirements for the twoversions. This trial-and-error experimentation, by which learning coach41 learns to make decisions about structural changes in student learningsystem 11, is similar to the trial-and-error process of estimatingcustomized hyperparameters illustrated in FIGS. 2 and 3.

The student learning system 11 and learning coach 41 can be implementedwith computer hardware and software. For example, they could be part ofan integrated computer system (e.g., a server or network of servers)that has multiple processing CPU cores. One set of cores could executethe program instructions for the student learning system 11, another setfor the learning coach 41, and so on. The program instructions could bestored in computer memory that is accessible by the processing cores,such as RAM, ROM, processor registers or processor cache, for example.In other embodiments, the student learning system 11 and/or the learningcoach 41 could execute on graphical processing unit (GPU) cores, e.g. ageneral-purpose GPU (GPGPU) pipeline. GPU cores operate in parallel and,hence, can typically process data more efficiently that a collection ofCPU cores, but all the cores execute the same code at one time. Thus, ifthe student learning system 11 and/or the learning coach 41 wereimplemented with a GPGPU pipeline, the GPU cores would need to taketurns executing the code for each system. In other embodiments, thestudent learning system 11 and/or the learning coach 41 could be part ofa distributed computer system. For example, the computer devices (e.g.,servers) that implement the student learning system 11 and learningcoach 41 may be remote from each other and interconnected by datanetworks, such as a LAN, WAN, the Internet, etc., using suitable wiredand/or wireless data communication links. Data may be shared between thevarious systems using suitable data links, such as data buses(preferably high-speed data buses) or network links (e.g., Ethernet).

FIG. 4 is a block diagram of an illustrative embodiment of the inventionaccording to other embodiments. In this illustrative embodiment, thereis a plurality of learning modules 101 and one or more data selectionmodules 105, all connected through a switching network 104. Eachlearning module comprises a large learning system 102 and a learningcoach 103. The large learning system 102 and the learning coach areexamples of the student learning system 11 and learning coach 41described above. As such, each large learning module 101 may be a deepneural network with up to one hundred or more layers, millions of nodes,and billions of connections. The large learning systems 102 may be, forexample, pattern classification systems, representing their output byselecting the best matching category or by computing a score for thedegree of match for each category. Alternately, the large learningsystems 102 may be regression type systems, representing their output asan approximation to some unknown function estimated from noisy samples.In one illustrative embodiment, under the guidance of the leachingcoaches 103, each learning module 101 is trained to specialize,performing best on particular problems or on particular kinds of data.

Each data selection module 105 may comprise a source of data 108, a datapreprocessor 107, and a learning coach 106 for the data preprocessor. Acopy of each item or batch of data 108 is converted to a packet and issent by the switching network 104 to one or more of the learning modules101, or is discarded. In the illustrative embodiment, each dataselection module 105 determines the destination of each of the copies ofeach packet of data 108. In one illustrative embodiment, the datapreprocessor 107 is a pattern recognition system performing someclassification task on the same data 108 that is to be sent to thelearning modules. As such the data preprocessors 107 and the learningcoaches 106 are examples of the student learning system 11 and thelearning coach 41 described above. That is, the learning coach 106 cancontrol the hyperparameters, structure or data flow of the datapreprocessors 107 as described herein. In another illustrative example,the data preprocessor 107 determines the destination of the data packets108 based on metadata. For example, in a speech recognition system, thedata preprocessor 107 may determine the destination of a packet based onthe known or estimated identity of the speaker.

As described herein, the learning coaches 103, 106 in FIG. 4 can, amongother things, change the architecture of an associated machine learningsystem 102, 107 by, for example, adding or deleting nodes in a network.It may control the learning process of the learning systems 102, 107 by,for example adjusting the step size in a stochastic gradient descentalgorithm, or the size of a batch for batch update. The learning coaches103, 106 may also impose a regularization or change the weight in aregularization of their associated student learning system 102, 107.When there is more than one associated machine learning system, one ormore learning coaches 103, 106 may supervise and control the combiningof knowledge of the machine learning systems or may combine the outputfor a particular data item. The learning coaches 103, 106 may evenchange the number of conventional machine learning systems workingtogether on a task, for example, by cloning one or more existing systemsand then training them differently, or alternately by combining two ormore systems into a single system. These actions by a learning coach maybe optimized by testing and evaluating the choices available to thelearning coach based on practice data, as described in the following USpatents, which are incorporated herein by reference in their entirety:U.S. Pat. Nos. 8,014,591; 8,180,147; 8,331,656; and 8,331,657.

In an embodiment in which the data preprocessor 107 is a patternrecognition system, the data preprocessor 107 may be similar to thelarge learning systems 102 in the learning modules 101 or it may becompletely different. In particular, in some embodiments it may be muchsmaller and simpler than the large learning systems 102. It may alsotake a completely different form. For example, the large learningsystems 102 may be deep neural networks, while the data preprocessors107 may be hidden Markov process models, random forests, or simplek-means classifiers. In some embodiments, either the large learningsystems 102 or the data preprocessors 107 may comprise a mixture ofmultiple types of pattern recognition systems.

Two or more large learning systems 102 may be combined into a single,larger learning system (which may be operating as a large learningsystem 102 in a larger overall system). The recent success of large deepneural networks has shown that, in applications with a very large amountof data, the larger the neural network the better the performance, up tosome limit that depends on the amount of data. However, in manyapplications there is a practical limitation on the size of a neuralnetwork that can be implemented efficiently. There are several ways toimplement the algorithms for deep neural networks and their training torun in parallel on many processor cores. It is especially cost effectiveto implement them to run on graphics processing units (GPUs) withthousands of processing cores on a single chip. With such a large numberof processing cores on a single chip, the limiting factor on the size ofa neural network implemented on a plurality of GPU boards is usually thedata communication bandwidth between the GPU boards.

In the illustrative embodiment in FIG. 4, each large learning module 101may contain one or more deep neural networks, or other learning systems,that are implemented on a single GPU board, or on a cluster of GPUboards that are connected by a very high speed data communicationchannel. In this illustrative embodiment, the data communicationbandwidth between the modules is assumed to be substantially slower thanthe internal communication bandwidth among components within a module.

Moreover, in the illustrative embodiment in FIG. 4, like the embodimentshown in FIG. 1, each large learning system 102 can be trained in aspecialized task, under the control and guidance of its learning coach103. In addition, the task for a specialized module 101 may be sospecialized that it only applies to a small fraction of the data, so aparticular specialized module might only be active a small fraction ofthe time. One illustrative embodiment of a plurality of modulesimplemented on a single computer or a single local area network, whichis shown in the exemplary embodiment of FIG. 5, would be to have most ofthe large learning systems 102 in an idle mode, perhaps just stored onsecondary storage (e.g., not directly accessible by the CPU, typicallyimplemented with hard disk drives, optical storage device and/or flashmemory) with a smaller number of modules active or in a ready state in acache in high speed memory (e.g., memory directly accessible by the CPU,such as main memory RAM, processor registers and/or processor cache).Specific examples of this embodiment will be discussed in the examplesbelow.

As an illustrative example of an application for such a hierarchy ofactive and idle modules, consider a system for monitoring and guidingfitness exercises. With instrumented exercise machines, the system wouldknow the exercise being performed by a particular individual, the musclegroups involved, the weight being moved, the number of sets to beperformed, the number of repetitions per set, the range of motion, therate of motion, the current point in the range, the steadiness ofmovement, and the entire past history of the individual. In someembodiments, the learning system might also have a model for many hiddenor latent variables, perhaps even a detailed model of cell physiology,neuro-muscular activation, and the cardio-pulmonary system. The systemcould recognize and track the state of each client as a function of timethroughout the exercise period. It could provide this information to apersonal trainer, or could directly provide advice and encouragement tothe client doing the exercise.

In this exercise and fitness monitoring example, there might be anindividually customized learning module 101 for each client. In theillustrative example, there might be a computer system in each gym orfitness center. However, only a small fraction of the users of anyparticular fitness center would be using it at any one time. The datafor the learning module for a specific user could be in secondarystorage when the user is not at the center.

Moreover, the system could be implemented across numerousgeographically-different fitness centers (tens, hundreds or thousands offitness centers) that are connected over the internet. The totalnetwork-wide system would collectively learn the optimum fitnessregimes. In this example, a fitness regimen would specify the types ofexercise, the number of sets and repetitions, and the progression ofincreasing weight as a client exercises over a period of weeks or more.To learn the optimum regimens, the system would learn from theperformance progression for a large number of clients with varyingregimens across the numerous fitness centers.

Every person is different from each other person. Each person differsfrom one exercise session to another. Therefore, there could be acustomized learning system (e.g., learning system 102 a in FIG. 4) foreach client. However, to learn optimum regimes, data needs to becombined from a large number of clients with similar characteristics andsimilar needs. In the illustrative embodiment, the data selectionmodules 105 could perform pattern recognition on the clients to groupthem together into groups who will be modeled together and who willreceive similar recommendations for training regimens. Active athletescould be treated differently from cardiac rehab patients, etc. Eachperson could also be treated differently at different stages in theirprogram as they become more fit. Thus, there is a need to customize toeach individual, but also a need to identify particular groups and learnshared properties of the members of each group. Finally, there couldalso be large learning systems 102 that try to learn universals thatapply to all people. The data selection modules 105 could send data fora particular person to the learning coach system 103 customized to thatindividual and also to the fitness regime learning module for the groupof users that is similar to the individual, and also to the moduleslearning knowledge that should apply to any user.

In an illustrative embodiment of the hardware systems supporting thispersonalized fitness system, there might be a dedicated computer with aGPU board in each gym or fitness center, and one or more data centersfor the modules learning the network-wide knowledge, such as optimumregimes and long-term fitness strategies. In a single fitness center,the individually customized learning modules could be in secondarystorage, except for the clients who are actually in the fitness center,whose learning modules would be in high-speed memory.

As a further illustrative example, consider handwriting recognition.There are two very different kinds of handwriting data: optical imagesof handwriting previously produced; and handwriting on a tablet or otherdevice that measures position as a function of time and that can alsoproduce a plot similar to an optical image of the writing. For theillustrative example, assume that the system is to recognize handwritingas it is being produced on a tablet. Because this mode also produces theequivalent of an optical image, the training can be bootstrapped fromoptical handwriting recognition data. If the handwriting to berecognized is produced on a tablet or smart phone, it is natural thatthe recognition be optimized to recognize the handwriting of the ownerof the mobile device. However, it will also be desirable for the deviceto recognize the handwriting when the owner has someone else writesomething on the device.

In one such embodiment, a recognition app might be running on hundredsof millions of devices (e.g., tablets), with a separate customizedrecognition module on each device and many, less customized, modulesrunning in data centers. Since each individual's handwriting is unique,there is the same need for customized, semi-customized, and universallearning systems as in the fitness center example.

In another embodiment of the handwriting recognition, each learningmodule 101 can be specialized in a completely different way. In thisembodiment, each data preprocessor 107 can try to recognize each writtenword, the same as the large learning systems 102. However, in thisillustrative embodiment, the objective of each data preprocessor 107 isnot necessarily to get the correct answer as the first choice, butrather to always include it in a short list of high scoring answers.This short list of answers, is then used by the learning coach 106 todirect the data for this particular recognition task to a learningmodule 101 that specializes in discriminating among the items in thisshort list. The learning coaches 103 could also group together thedifferent short lists in various ways. For example, all short lists withthe same top choice may be grouped together, resulting in one group foreach word in the vocabulary. In another example, for each word in thevocabulary, there may be a different group for each pair of possiblewords that occur in the short list. In that example, there would bedifferent embodiments depending on the number of copies made for eachdata item. In one embodiment, the group would depend on only the top Nscoring word choices (e.g., N=2) and the only copy of the data would besent to the learning module for that group of N words. In anotherembodiment, there could be a separate copy of the data item sent to aspecialized learning module specializing in each of the pairs of wordsthat occur on the short list. Thus, among these embodiments, the numberof learning modules will range in size from the size of the vocabulary,which may be 100,000 words or more, up to 100 times the vocabulary sizeor more.

The learning coaches 103 for the learning modules 101 may combine groupsof learning modules 101 to create a smaller set of groups. In someembodiments, however, they may have more than one physical moduleassigned to the same specialized task. Such an embodiment might be used,for example, if the learning modules 101 were running on a cooperativenetwork of end-user systems. Then, even if many of the physical systemsare offline at a particular time, there would usually be one or morebackup systems that could do the same specialized task.

These word-specific embodiments with 100,000 to 10,000,000 modules alsoillustrate another aspect of embodiments of the invention. Not all wordsare equally likely, and for any particular word only a few other wordshave a high probability of being confused with the particular word.Therefore, in these embodiments, some of the word-specific learningmodules will only infrequently receive data from the data selectionprocess. During training, these less frequently used modules can be keptin secondary storage and only be loaded into high speech memory whenenough data specific to a particular module has been assembled tocomplete a batch update in a stochastic gradient descent, as illustratedin FIG. 5. Stochastic gradient descent with batch update is a commonmethod for training deep neural networks and is well known to thoseskilled in the art of deep neural networks. In this embodiment, thetotal size of the models in secondary storage can be several orders ofmagnitude larger than the size of high speed memory.

If recognition is done offline rather than in real time, the recognitioncomputations can be sorted into batches by the switching network 104 andthe staging of modules from secondary storage to high speed memory bythe learning coaches 203 can be organized so that the necessary modules101 are active for each batch of recognition computations. In manyapplications, the amount of computation needed for recognition is muchless than for training because the training data must be presented manytimes to reach convergence in stochastic gradient descent trainingalgorithms such as are used for training deep neural networks. Thus,fewer computations are needed for real time recognition than are neededfor efficient training on large training sets. Thus, in some embodimentsthat need real-time recognition, the recognition may be implemented ondifferent hardware, such as clusters of CPUs with very large RAM, ratherthan on GPUs. Such a CPU cluster can also have a large amount ofsecondary storage. Since a CPU cluster is much slower than a cluster ofGPU boards, and since the CPU cluster typically has very large RAM as acache, the occasional loading of some modules from the secondary storagewill be better able to keep up with this implementation of real-timerecognition.

Another example of the embodiment illustrated in FIGS. 4 and 5 isautomatic speech recognition. Since every voice is different, automaticspeech recognition modules 101 could be specialized to individualvoices, as with the personal fitness systems or with the handwritingrecognition. As another example, some of the modules 101 could bespecialized to each word in the vocabulary or to short lists of highscoring words. This embodiment would be similar to the word-specificspecialized modules discussed above for handwriting recognition.

As an additional example, speech recognition could be organized aroundthe sequences of sounds and how the sounds vary as a function ofcontext. The pronunciation of each sound is heavily influenced by thepreceding and following sounds. One embodiment of the invention forspeech recognition based on sound sequences would be to have aspecialized learning module 101 for the middle sound in each possiblesequence of three sounds. In this embodiment, the sound sequence wouldbe recognized by the data preprocessors 107 and the data for the middlesound would be routed by the data selection learning coaches 106 and theswitching network 104 to the learning module 101 that specializes in thesound triple that has been recognized by the data preprocessor 107. Thelarge learning system 102 in the designated learning module 101 does notmerely rescore the putative sound sequence, but performs a newrecognition from scratch. During training it receives data for everyinstance in which the data preprocessors 107 recognize its particularsound triple, so it learns not only the characteristics of that soundsequence, but also of every sound sequence that the preprocessors 107misrecognize as that sequence. Note that this embodiment specializesbased on the sound triple as recognized, not the sound triple ashypothesized by the large learning system 101. This embodiment is incontrast to the triphone models that are well known to those skilled inthe art of speech recognition using a beam search with a hidden Markovprocess model. The triphone models in such a beam search are specific tothe triple of sounds hypothesized on a particular path in the beam, noton a triple of sounds as recognized by a data preprocessor 107. Thisdifference forces the training of a specialized module 101 to compensatefor the errors made by the preprocessor, adding robustness. Moreover, atrecognition time, there is much less computation because the learningcoaches 106 direct each data packet to only one or a small number oflearning modules 101.

In one embodiment of the sound sequence methodology, the unit of soundcould be a phoneme, which has been a common sound unit for largevocabulary speech recognition systems. However, the concept of “phoneme”is a linguistic abstraction, grouping together different sounds that arenot distinguished by the speakers of a particular language even if thosesounds might be distinguished in some other language. In anotherembodiment of the invention, however, the unit of sound would be theallophone, a narrower, more precise transcription of the actual sound asspoken. This embodiment would make it easier to share knowledge aboutthe sounds from one language to another. It would fit well with theembodiment in which the learning modules 101 would specialize on triplesof sounds as recognized by the data preprocessors 107 rather than ashypothesized by a higher level search. In particular, many of the datapreprocessors 107 and specialized learning modules could be sharedacross languages. Some languages might have some extra allophones andsome allophones that do not occur, but many of the allophones and theirsequences could be shared. If two languages have some differences intheir phoneme sets, those differences could be reflected in how theresults from specialized allophone-based large learning modules 102 arecombined for decisions at the phoneme or word level.

Another aspect according to some embodiments can be illustrated by theexample of speech recognition, interpolation of specialization along acontinuum. Let module A and module B be two specialized modules 101,with the training set T_(A) for module A and training set T_(B) formodule B. For example, module A and module B may be specialized to eachof two different speakers, or they may be specialized to two differentlanguages or two different dialects. Then a new specialized module (A,B, γ) may be formed anywhere along a continuum between A and B by usinga random selection of training data from sets T_(A) and T_(B), where theprobability of choosing an item of data from set TA is (1−γ) and theprobability of choosing an item of data from set T_(B) is γ.

A large number of interpolations along the continuum from A to B may betrained and stored on secondary storage as illustrated in FIG. 5.Another aspect of this embodiment is especially apparent if the modulesA and B have identical network architectures and they are specialized totwo similar speakers, two similar languages, or are otherwise similar toeach other. For example, two similar languages will differ in only a fewphonemes; two similar speakers will agree on the pronunciation of mostwords, and so forth. The mappings of the phoneme set or the allophoneset and their associated acoustic features will mostly vary continuouslyalong the continuum of interpolated modules. When two interpolatedmodules along the continuum have only a small difference, thenregularization may be applied in their training to make the trainingmore resistant to overfitting and to make the models vary more smoothlyalong the continuum.

However, there must be a few discontinuities if one of these similar butdistinct discrete sets is different in module B than in module A. Nullhypothesis testing (as described in U.S. Pat. No. 8,014,591) by thelearning coaches 103 for module A and module B can determine where alongthe continuum is the best place to break the regularization and allowthe models on the two sides of the break to be different. Thus, with theguidance of the learning coaches, the system will be able to learnexplicit structural knowledge as well as the usual parametric knowledge.

FIG. 6 is a block diagram of another illustrative embodiment. In thisembodiment, the learning coaches 303 enable the deep learning systems102 to represent and learn knowledge in an explicit form that can beinterpreted by humans and by other systems. In large learning systems,such as deep neural networks with many layers or even stochastic modelswith many interdependent latent or hidden variables, the knowledgeacquired during learning is often represented in the interactions amongthe many elements in the system. In that case the knowledge is difficultor impossible to interpret and is inaccessible not only to human usersbut also to other computer systems. These other computer systems may beother learning modules cooperating on the same task or they may besystems controlling stationary or mobile equipment.

One attribute that is different in the embodiment illustrated in FIG. 6from the embodiment illustrated in FIG. 4 is that the embodiment in FIG.6 applies even when there is only one large learning system 102. Anotherelement that is different in the embodiment illustrated in FIG. 4 is thesupport system 307 with explicit representation of knowledge, ratherthan the data preprocessor 107 shown in FIG. 4.

An important property of explicit knowledge is that it can be morereadily communicated than implicit knowledge. It is more interpretableby humans and can more easily be transferred from one learning module101 to another, or from a learning module to another computerapplication such as a control system.

FIG. 6 is a block diagram of an illustrative embodiment in which alearning coach 303 extracts explicit knowledge from the support system307 and introduces the explicit knowledge into the inner layers of adeep learning network 102 by any of several means, as well as optionallypresenting it to humans 311 and sharing it with other systems 312. Inthe illustrative embodiment, the learning coach 303 presents theknowledge to the deep learning network 102 in one of two forms. One formin which the learning coach 303 can present knowledge is as a set ofnodes and activation levels for those nodes associated with a particularset of data input values. In some embodiments, particularly if thesupport system 307 is also a deep learning network, the learning coach303 may present knowledge to the learning network 102 in the form ofconnection weights between particular nodes. The methods that thelearning coach 303 uses to associate particular nodes in the presentedknowledge with particular nodes in the inner layers of the deep learningnetwork will be discussed in association with other figures.

FIG. 7 illustrates one embodiment by which a learning coach 303 mayinject explicit knowledge into an inner layer of a deep neural network102 (see FIG. 6). This illustrative example assumes that the explicitknowledge is represented in the form of activation values for aspecified set of external nodes. The nodes may either be a set of nodeswithin the support system 307 shown in FIG. 6 as selected by thelearning coach 303 or a set of nodes within the learning coach 303 thatrepresent knowledge extracted from the support system 307. The nodeactivations may depend on the data input to the deep learning network102 and may also depend on external data that is not available to thedeep learning network 102.

The learning coach can select a particular inner layer 402 of the deeplearning network as the layer to receive the injected knowledge. Thelayer 402 may have thousands of nodes and there may be many moreconnections than the few that are shown. The learning coach can make avirtual copy 401 of the specified set of external nodes and connect thevirtual nodes to the selected layer 402 and to the lower layers 404. Thenodes in the selected layer 402 receive input from the virtual nodes401, but do not back propagate to the virtual nodes 401 during training.The virtual nodes 401 can back propagate through the connections to thenodes in the lower layers 404, but their activations can be controlleddirectly by the learning coach, so they do not receive input from thelower layers.

In some embodiments, the learning coach 303 can control the dropout ratefor the virtual copied nodes. Dropout is a well-known technique intraining deep neural networks. It consists of performing training stepsin which certain nodes are forced to have a neutral value or to beunobserved, regardless of what their activation levels may be for thecurrent data being processed. Dropout generally trains a deep neuralnetwork to be more robust because training in the presence of dropoutforces the neural network to learn to estimate the missing data usingother paths through the network. In the embodiment illustrated in FIGS.6 and 7, the objective of the learning coach 303 can be to coach thedeep learning network 102 to learn the explicit knowledge. In someembodiments, the total system is designed to have the external supportas a permanent addition to the deep learning network. In otherembodiments, the system is designed for the deep learning network 102 tolearn to estimate the explicit knowledge for itself and to eventually bedisconnected from the support system 307. In either case, the learningcoach 303 preferably monitors the performance of the network 102 towardsthe specified objective. The learning coach can then optimize thedropout rate and other control parameters based on null hypothesistesting on practice data, as described in U.S. Pat. No. 8,014,591, whichis incorporated herein by reference in its entirety.

FIG. 8 illustrates another variant of the embodiment illustrated in FIG.7. In this variant another set of nodes 505 is added to the selectedlayer. These added nodes 505 are in a one-to-one relationship with thevirtual nodes 401 and a regularization is applied to make theiractivations tend to agree with the corresponding virtual nodes.Regularization is a well-known technique to those skilled in the art ofstatistical estimation that smooths statistical estimates and makes themmore robust. In this case, the regularization consists of an additionalterm in the objective function during training that penalizesdifferences between each node in set 505 and its corresponding virtualnode in set 401. The regularization and the respective dropout rates ofthe virtual nodes 401 and the regularized nodes 505 are all controlledby the learning coach 303, with an objective that is optimized bytesting on practice data.

In this optimization, the learning coach 303 may use an objective thatis different from the performance objective of the deep learning network102. For example, the learning coach may be optimizing theregularization and the dropout rates primarily to make the knowledge inthe deep learning network 102 more explicit while the deep learningnetwork optimizes for minimum error rate.

FIG. 9 is a flow chart of an illustrative embodiment of the supportsystem 307 of FIG. 6 with a block diagram of its relationship to thelearning coach 303 of FIG. 6 and the deep learning network 102 of FIG.6. At step 618, the support system 307 under control of the learningcoach repeatedly acquires external data and, at step 608, acquires asuccession of instances of input data to the deep learning network innerlayer 402. Note that the block 402 in FIG. 9 represents the same thingas the inner layer 402 shown in FIGS. 7 and 8. External data at step 618is data from some source other than the input data to the deep learningnetwork at step 608. It may be data that does not change, or that changeslowly rather than for each item of input data. It may be data that ismeasured by other sensors. It may be present for some items of inputdata and not present for other. For example, in some embodiments of someapplications, external data may be in the form of metadata that is inthe header of a data packet that contains the primary data to berecognized. An example of metadata would be the source and destinationfor a phone call, or the sender and the addressee of an email. In theillustrative example of a personalized fitness program, measurementssuch as heart rate or blood pressure may sometimes be monitored but maynot always be available. They may be external data in some embodiments,but may be input data in others.

Each instance of data is processed by the support system 307 to produceknowledge in a form that the learning coach 303 can use to inject theknowledge into a deep learning system 102. For the embodimentillustrated in FIGS. 7 and 8, this knowledge is represented as theactivation values for a set of nodes. Depending on the type of patternrecognition or modeling that is being performed, the system chooses adifferent path at decision point 613. If the system is doingclassification into a finite set of categories, then the control flowproceeds to block 614, which classifies the input data, perhaps alsoutilizing the external data. The output of block 614 is either a scorefor each possible classification category or simply an indication of thebest matching category, which is equivalent to a score of 1 for thechosen category and 0 for everything else. Each category is associatedwith a node in the set 401 of FIGS. 7 and 8 and the corresponding scoreis the activation value for the node. The scores computed by the patternclassification 614 may be transformed by the learning coach 303 to makethe scores compatible with the activation values used in the deeplearning network 402.

If the support system is not doing classification into a discrete,finite set, then the flow of control proceeds to block 615, whereanother type of pattern recognition or model estimation is performed,and block 616, where latent variable from the pattern recognition ormodel estimation from block 615 are encoded as node activations. Flowgoes in this direction, for example, if knowledge to be transferred isin the form of continuous-valued variables, rather than classificationinto a finite number of categories. In the exercise and fitness trainingapplication, blood pressure, heart rate, and rate of motion are all inprinciple continuous-valued measurements. Even if they might be roundedoff to integer values in external measurements, if they are estimated bya pattern recognition system, or just smoothed by filtering they willvary as a function of time and will take on fractional, non-integervalues to fit a smooth curve. Continuous-valued variables would alsoresult if the pattern analysis system estimates a regression functionrather than classification categories. Such continuous-valued variableswill be coded as values in a finite set by breaking up the range ofpossible values into a set of intervals. The intervals may bepredetermined, or they may be specified by the learning coach 303.

In some embodiments, the explicit knowledge to be transferred may justbe external measurements, such as blood pressure readings without anysmoothing or re-estimation in the support system. In an embodiment inwhich all of the knowledge to be transferred is already known fromexternal measurements, then the support system would not need to do anypattern recognition or modeling. However, even in that case, the set ofvalues of the external measurements might be mapped into a finite set ofnodes representing intervals in the range of possible values.

In another aspect, rather than performing pattern recognition per se,block 616 may build and estimate a model for phenomena underobservation. For example, in the exercise and fitness application, themotion of the client's body and of the weights in an exercise machinemay be modeled by Newton's laws of motion. As another example, in aspeech recognition task, block 616 may model the speech as a hiddenMarkov process or as a dynamic Bayesian network, modeling techniqueswhich are well-known to those skilled in the art of speech recognition.Speech might even be modeled as a many-layered stochastic process withlayers for raw acoustic features such as amplitude as function of timeand frequency, derived features such as fundamental frequency andformant frequencies, linguistic distinctive features, articulatoryfeatures, allophones, phonemes, diphones, triphones, demi-syllablesincluding onsets and codas, syllables, words, phrase, clauses,sentences, paragraphs, dialogues, documents, books, and libraries. Someof the estimated hidden or latent random variables may becontinuous-valued.

In such models, whether stochastic or deterministic, continuous-valuedvariables would be represented by a finite number of intervals asdescribed above. In the Markov process model and in the Bayesian networkmodel, the states in the Markov process or the nodes in the Bayesiannetwork would correspond to the nodes to be cloned in virtual nodes 401(see FIGS. 7-8).

Another aspect of performing the steps of blocks 615 and 616 occurs ifmore than one variable is being estimated, whether the individualvariables are discrete-valued or continuous valued. Eachcontinuous-valued variable could first be coded as a discrete-valuedvariable, as described above. Then, the learning coach 303 could createa separate set of virtual nodes for each variable.

Another aspect deals with the issue of determining the activation valuesto be used for nodes produced by blocks 615 and 616. If the nodesrepresent intervals of a deterministic variable, then only the nodecorresponding to the value of the variable would be activated. However,if the nodes represent states in a hidden stochastic process orintervals for an estimated random variable, then the node activationswould represent some form of probability distribution. If the dataobservations are made as a function of time, then the activation valuesmight represent either joint probabilities or conditional probabilities.The activation probabilities might be conditioned on (or joint with)either the past or the future, or both. In some embodiments, the nodeactivations might be the probabilities themselves, perhaps normalized tosum to one across the nodes in a given set. In some embodiments, theactivations might represent logarithms of probabilities, which wouldmake them more compatible with other measurements such as distancesbetween points and clusters. The choices among these alternatives wouldbe controlled by the learning coach 303, which in some embodiments wouldperform automated testing on practice data.

Finally, the nodes and activations produced as described above could becloned and passed to the deep learning inner layer 402 as described inthe discussion of FIGS. 7 and 8.

Another aspect of the present invention deals with making implicitknowledge explicit and communicable. The support system 307 can bespecifically designed to make its knowledge explicit and communicable asillustrated in FIGS. 7-9. After the processes illustrated in FIGS. 7-9have been performed, the deep neural network 102 (see FIG. 6) will alsohave some knowledge that is explicit and communicable. A deep neuralnetwork that has been trained without the processes of FIGS. 7-9 mayhave a great deal of knowledge. Such deep neural networks have brokenprevious records in many benchmark tests.

However, this knowledge is implicit and is often difficult or impossibleto interpret. For example, even two learning modules 101 with deepneural networks 102 having identical architectures, after many rounds oftraining to specialize them to different tasks as described in thediscussion of FIG. 4, may have nodes with completely different patternsof activation when presented with the same data. These sibling moduleswould be unable to communicate their knowledge to each other. Even adeep neural network that has been coached to have explicit knowledge,such as by the procedures illustrated in FIGS. 7-9, will also have agreat deal of implicit knowledge in addition to the explicit knowledge.

FIG. 10 illustrates embodiments of several methods for making suchimplicit knowledge explicit and communicable between machine learningsystems and more easily interpretable by humans. Two forms of explicitknowledge are illustrated: (i) nodes that explicitly represent featuresand (ii) clusters of related data examples. The feature node knowledgeand cluster knowledge can be trained jointly, and they can interact andsupport each other both during training and during operational use.Several illustrative means of communicating and sharing the knowledgeare shown. For example, there are two forms of soft tying nodeactivations that will be explained below. In addition, once featurenodes or clusters have been trained, say in machine learning systems1023 and 1024 respectively, the knowledge may be used for supervisedtraining of other systems such as machine learning systems 1025 and 1026respectively. This supervised training can be done in either direction.For example, feature detector 1023 can be used to train feature detector1025, or feature detector 1025 can be used to train feature detector1023. Similarly, cluster classifier 1024 could be used to train clusterclassifier 1026, and vice versa. Finally, a set of one or more featurenodes can be inserted into another network, for example, the machinelearning system 1022, using the techniques described in connection withFIGS. 7 and 8.

There are six machine learning systems shown in the example of FIG. 10:Machine learning system 1021 with internal variables 1012; featuredetector system 1023 and 1025; cluster classifiers 1024 and 1026; andone or more other machine learning systems with explicit knowledge 1022.There can also be a learning coach 1011 that controls the iterativetraining of feature detector 1023 and cluster classifier 1024, as wellas controlling the soft tying of node activations among the systems aswell as internal soft tying of node activations within one or more ofthe machine learning systems.

Feature detector 1023 and cluster classifier 1024 may be trained bysupervised training. In an illustrative embodiment, however, featuredetector 1023 and cluster classifier 1024 are iterative trained withunsupervised learning, at least initially. If category labels are known,the data examples may be separated by category and then clusters aretrained unsupervised within each category. The unsupervised training isstarted by assigning each data example to a cluster by any convenientmeans, for example randomly. Each cluster is defined by the dataexamples that are assigned to it. In some embodiments, each data examplemay be assigned to multiple clusters with each assignment having anassociated weight (e.g., a confidence weight) such that the assignmentweights for each data example sum to one.

Given a set of assignments of data examples to clusters, selected nodeswithin machine learning system 1021 can be designated as potentialfeature nodes for one or more clusters. Each potential feature node ndesignated for a cluster has its activations values α_(n)(x) soft tiedfor all data examples x associated with that cluster. In this form ofsoft tying, an extra regularization term is added to the cost functionfor the potential feature node. For a data example x associated with thecluster, the regularization cost term can be based on the differencebetween the value α_(n)(x) and the average activation value averagedacross all data assigned to the cluster. For example, the soft tyingregularization can be the L2 norm, L2_(n)(x)=(α_(n)(x)−μ_(n))². Thevalue μ_(n) is the mean activation for node n over all of the dataassociated with the cluster. To save computation in some embodiments,this mean value is estimated from the mean value in the previousiteration.

In an illustrative embodiment, the training of the feature nodes andclustering is done iteratively. The features can be the output ofdedicated feature detectors, such as the feature detector 1023 or thefeature detector 1025, or they can simply be internal nodes in machinelearning systems, such as machine learning systems 1021 or 1022. Theiterative training can use any of many clustering algorithms that arewell-known to those skilled in the art of machine learning, for examplek-means clustering. In standard k-means clustering, each cluster isrepresented by the mean of the data examples assigned to the cluster. Ina step in the iterative process, each data example is reassigned to thecluster whose mean is closest to the data example. When each dataexample has been reassigned, the cluster means are recomputed based onthe new assignments. This iterative process is repeated until there areno changes in the assignments of the data examples or some otherstopping criterion is met.

An illustrative example of training a designated set of nodes to learnfeatures uses k-means clustering, with the data examples for theclustering using the activation values of the designated nodes insteadof or in addition to the input data 1001.

The illustrative embodiment shown in FIG. 10 is more general. The simplecomputation of the mean of the data examples assigned to the cluster isreplaced by a general purpose machine learning system 1024 or 1026 as acluster classifier. The output of either cluster classifier 1024 or 1026for a data example x is a set of scores with one score for each cluster.The distance between the data example x and the cluster mean, which isused in k-means clustering, is a trivial special case of the clusterclassification scores that can be computed by cluster classifier 1024 or1026, which are much more general. Either cluster classifier 1024, 1026can approximate an arbitrary non-linear function. For example, thecluster score can represent a non-convex region or even a region that isnot a connected set.

In addition, the input to cluster classifier 1024 or 1026 is not limitedto the node activations of a set of designated feature nodes. In theillustrative embodiment shown in FIG. 10, blocks 1023 and 1025 arededicated feature detectors. Again, simply copying the node activationsof a set of designated feature nodes is a trivial special case of thefeatures that can be computed by feature detector 1023 or 1025. Inaddition, the feature detectors 1023 and 1025 and the clusterclassifiers 1024 and 1026 can have access to additional informationbesides the input data and the node activations of a designated set ofpotential feature nodes.

In addition to its normal output 1031, the machine learning system 1021can also have a set of extra output nodes 1032. These extra output nodescan be trained by back propagation from cluster classifier 1024 andfeature detector 1023 to compute any useful information that can becomputed from the internal variables 1012 of the machine learning system1021. For example, if machine learning system 1021 is a deep neuralnetwork, these internal variables include the node activations of all ofthe inner layer nodes as well as the input and computed output values.In addition, during training these internal variables include thepartial derivatives of the cost function with respect to each of thenode activations and with respect to each of the connection weights andany other learned parameters.

In addition, nodes within feature detector 1023 and cluster classifier1024 can be soft tied to nodes within the machine learning system 1021,which in turn may be soft tied to nodes in feature detector 1025 andcluster classifier 1026. In one illustrative embodiment, featuredetector 1025 and cluster classifier 1026 are dedicated stand-alonesystems trained by supervised training and/or soft tying to systems,such as the feature detector 1023 and the cluster classifier 1024, orfeature detectors and cluster classifiers associated with one or moreother machine learning systems, such as machine learning system 1022.Soft tying of node activations between two or more nodes in separatemachine learning systems is the second form of soft tying mentionedabove. In this form of soft tying, the node activations being tied arein separate machine learning systems, such as systems 1021, 1022, 1025and 1026, in which the forward activation in each machine learningsystem is processing the same input data example x in each of therespective inputs 1001, 1002, 1005 or 1006. In this form of soft tying,the relaxation cost is based on the mean of the activation values of allthe tied nodes for the same input data example x, rather than a meanacross data examples. This second form of soft tying may also be appliedto learned parameters, such as the connection weights in a deep neuralnetwork. Although only a few soft tying connections are shown with thedouble-headed dashed lines in FIG. 10, any two machine learning systemswith internal node activations or other latent variables may be softtied, including all of the machine learning systems shown in FIG. 10.The soft tying is bidirectional.

The feature detector 1025 and the cluster classifier 1026 may be trainedby supervised learning from feature and cluster assignments learned byfeature detector 1023 and cluster classifier 1024. On the other hand,feature detector 1025 and cluster classifier 1026 may be trained by oneor more other machine learning systems, such as machine learning system1022. In that case, the roles may be reversed and in some embodiments,feature detector 1023 and cluster classifier 1024 may be trained bysupervised learning from feature detector 1025 and cluster classifier1026.

When feature detector 1023 and cluster classifier 1024 are trained bysupervised learning, they can back propagate through the extra outputnodes 1032 of the machine learning system 1021, so again they trainmachine learning system 1021 to provide useful computations based on anyof the internal variables 1012 in machine learning system 1021.

In the illustrative embodiment, each instance of either form of softtying is implemented as an extra term in the cost function for each ofthe nodes that are tied. There are one or more hyperparametersassociated with each node in each instance of soft tying. For example,there is a relaxation strength hyperparameter that multiplies thedifference between the activation value of a node n for data example xand the mean value across the tied data examples or the tied nodes. Insome embodiments, this hyperparameter is customized according to thesituation. For example, in early phases of discovering the features andclusters for machine learning system 1021, many nodes may be designatedas potential feature nodes with little or no evidence yet available. Inthis case, the relaxation strength hyperparameter may be very small. Atthe other extreme, with a large value of the relaxation strength softtying may approximate hard tying, such as the sharing of statistics forcorresponding connection weights in a convolutional neural network.Thus, among other things, soft tying can implement a generalization ofconvolutional neural networks.

In some embodiments, feature detector 1023 and cluster classifier 1024are treated as part of machine learning system 1021 and remain activeand associated with machine learning system 1021 during operational useas well as during training. In other embodiments, feature detector 1023and cluster classifier 1024 are only used in training. The assignmentsof data examples to clusters are used in the soft tying of feature nodeswithin machine learning system 1021. In addition, these internal featurenodes are trained by back propagation through the extra outputs 1032 andby soft tying to nodes in feature detector 1023 and cluster classifier1024 and perhaps to one or more other feature detectors and clusterclassifiers, such as 1025 and 1026.

The control of this iterative training and of the associatedhyperparameters is a complex process. In the illustrative embodiment,this complex process is preferably controlled by learning coach 1011,which also controls the soft tying and associated hyperparameters.

By soft tying node activations and data clustering information, any ofthe learning modules 101 in FIG. 4 (or FIG. 11 described below) mayshare knowledge with any of the other learning modules 101. Thesemodules would then have shared explicit knowledge with knowncorrespondence between their respective cloned virtual nodes. Therewould also be a known correspondence between the associated nodesregularized to the cloned virtual nodes.

As a further aspect, the corresponding nodes in two or more deeplearning networks 102 regularized to virtual nodes cloned to the sameexternal knowledge representation node could in turn be regularized toeach other. This regularization could be maintained in further trainingeven when the cloned explicit knowledge is not present. This aspect isillustrated in FIG. 11, in which the large learning networks can beviewed either as large learning networks 102 as illustrated in FIGS. 4and 6, or as machine learning systems such as 1021 as illustrated inFIG. 10. Explicit knowledge is communicated as illustrated in FIGS. 7 to9 or as illustrated in FIG. 10.

When a pair of connected nodes in a learning network 102 or 1021 isregularized to a corresponding pair of nodes in another learningnetwork, their connection weight may also be regularized, subject totesting of the effectiveness of the connection regularization by thelearning coach. Regularization of the connection weights may beespecially effective if an entire set of source nodes are regularizedand an entire set of destination nodes are regularized and if there arevery few other source nodes connected to the destination nodes. Then allthe connections between the source nodes and the destination nodes maybe regularized.

This cross-module regularization enables two or more learning modules toshare knowledge over a data communication channel even with the lowerbandwidth that would be available outside a GPU cluster. In mostembodiments, only a small fraction of the nodes would represent explicitknowledge and only a fraction of those would be regularized acrossmodules. Furthermore, regularization is only a weak link and would notrequire communication with every data item for nodes or every update forconnection weights.

As a further aspect of the present invention, this sharing of knowledgedoes not have to be communicated all at once to the entire collection oflearning modules 101. Instead, as described in U.S. Pat. No. 8,331,657(which is incorporated herein by reference) knowledge could at first beshared with a few other modules. The knowledge could be tested by thelearning coaches to see if it improves the performance of those othermodules. Knowledge that did not improve the performance of an adequatefraction of the testing system could be rejected. Knowledge thatimproved the performance of many of the testing systems could be passedon to additional systems.

In some embodiments of the invention, the student learning system 11 isan ensemble of machine learning systems rather than a single machinelearning system, as is shown in FIG. 12. One illustrative embodiment ofthe learning coach 41 making a structural change to student learningsystem 11 avoids making a decision as to which version of the structureshould be used for continued training. In this illustrative embodiment,both the original ensemble component 801 b and the modified ensemblecomponent 801 c are retained as members of the ensemble, creating anensemble with two members from a single machine learning system, orincreasing the number of members in a machine learning system that isalready an ensemble. Viewing the whole ensemble as the student learningsystem 11, starting with the case of a one member ensemble, adding amember to the ensemble is just an instance of making a structural changein student learning system 11. This instance of a structural change istreated like any other structural change.

The ensemble student learning system 11 as in FIG. 12 also illustratesanother aspect of the invention according to various embodiments, whichis the ability of learning coach 41 to control the selection of data 821for student learning system 11. In this illustrative embodiment, thelearning coach 41 selects different subsets of the data to send todifferent members of the ensemble, by sending appropriate controlsignals to the data switching network 831. However, the student learningsystem 11 of FIG. 12 is only one illustrative embodiment. The embodimentmay also be used with any ensemble machine learning system, includingensembles that include a heterogeneous mixture of machine learningcomponents of different types.

In this illustrative embodiment, the data selection is done by apreliminary classifier 803, as shown in FIG. 13. The preliminaryclassifier 803 is trained under the guidance of the learning coach 41,for example. The preliminary classifier 803 may have a major favorableimpact on the computational cost of the ensemble as a whole. Forexample, in a very large ensemble, with thousands or millions ofcomponents, the preliminary classifier 803 may select only a very smallfraction of the ensemble components to receive each data example, bothin training and in testing and operational use. The amount ofcomputation for either training or operation tends to grow at leastproportional to the amount of data. Therefore, with a large ensemble,the preliminary classifier 803 may be able to reduce the amount ofcomputation by orders of magnitude. In the illustrative embodiment, thepreliminary classifier 803 is not trained simply as part of the ensemblesystem. For example, even if the preliminary classifier 803 and all theensemble components 801 a-d are neural networks, the preliminaryclassifier 803 is not trained merely by back propagating the derivativeof an error cost function back through the ensemble members 801 a-d andthen through the preliminary classifier 803, as might be done for onelarge combined network. Instead, the preliminary classifier 803 cantrained directly by the learning coach 41, for example, to optimize acombination of the cost of errors and the cost of computation for theensemble as a whole.

In the example illustrated in FIG. 12, a new ensemble member 801 c hasbeen created by the learning coach 41 by making a change in an originalensemble member 801 b, with both the old member 801 b and the new member801 c retained in an expanded ensemble. If the difference between 801 band 801 c is small, then there may be a significant increase in theamount of computation without much reduction in the error rate.

However, the preliminary classifier 803 illustrated in FIG. 13 can bothreduce the amount of additional computation and help train ensemblemembers 801 b and 801 c to reduce the error rate. In this illustrativeembodiment, the preliminary classifier 803 would select different datato send to ensemble member 801 b than to ensemble member 801 c. Animportant property of preliminary classifier 803 is that it performsthat same preliminary classification on test data as on training data.

In one illustrative embodiment, the learning coach 41 tests both theoriginal ensemble member 801 b and the new ensemble member 801 c on aset of validation data. In this illustrative embodiment both 801 b and801 c are to be retained. The test on the validation data providestraining data for preliminary classifier 803. For each item of data,preliminary classifier needs to decide whether to distribute the item toensemble member 801 b, ensemble member 801 c, or both. In oneillustrative embodiment, the preliminary classifier 803 is trained withsupervision to try to predict which of the two ensemble members 801 b or801 c will have a higher score for the correct answer. It then sendsthat data item to the ensemble member that it predicts will have thehigher score.

This illustrative embodiment applies to a preliminary classifier 803used in combination with any ensemble of classifiers 801 a-d, whether ornot the ensemble of classifiers are obtained by the learning coach 41adding members to an ensemble by modifying existing members. For anyensemble of classifiers 801, a preliminary classifier 803 can be trainedby supervised learning to predict, for each member of the ensemble, theprobability that the given member of the ensemble will correctlyclassify a given input data item. In this supervised training, thepreliminary classifier does not classify the input data item and is nottold the correct category to classify the given input data item. Rather,in its supervised training, the preliminary classifier is merely toldwhich of the ensemble members successfully classify the given data.Thus, the preliminary classifier learns to predict the probability ofsuccess of the ensemble members, which is what it needs to know indeciding which ensemble members should be assigned each item of data.

Once an initial version of preliminary classifier 803 has been trained,an iterative process can be performed in which 801 b and 801 c areretrained, with each of them receiving as training data only the datathe preliminary classifier selects for each of them. Thus, ensemblemembers 801 b and 801 c will learn to specialize on the respective dataselected for each of them by preliminary classifier 803. Then theprocess is iterated, with ensemble members 801 b and 801 c again bothclassifying a shared validation data set, which is used to retrainpreliminary classifier 803, and the process is repeated untilconvergence or some other stopping criterion is met.

In the illustrative embodiment, the preliminary classifier 803 selectsone or more destinations for each data item. In this embodiment, thelearning coach 41 can control one or more locally optimizedhyperparameters that control the average number of ensemble members thatreceive each item of data. For example, let MaxScore be the best scorefrom the preliminary classifier, and let h be a hyperparametercontrolling the distribution of data. The learning coach 41 can set ascore threshold h, such that only destinations that get a preliminaryclassifier score s>MaxScore−h receive a given item of data, duringeither training or recognition.

FIG. 14 illustrates another aspect of the invention according to variousembodiments. In the embodiment illustrated in FIG. 14, the learningcoach 41 combines its ability 322 to control hyperparameters with itsability 332 to modify the structure of the student machine learningsystem 11. In this illustrative example, the learning system 11 is adeep learning neural network. The specific type of modification 332 tothe structure of student learning system 11 made by learning coach 41 isthe addition of one or more nodes or arcs to the network 11. In thecontrol of the hyperparameters 322 in the embodiment illustrated in FIG.10, the learning coach 41 controls the learning rate of the new parts ofthe structure differently from the learning rate for the older parts ofthe network in student learning system 11.

In the embodiment illustrated in FIG. 14, two issues are important. Whennew structure is added to a network that has already been extensivelytrained, the parameters associated with the new part of the network mayneed more training than the parameters associated with the older partsof the network. On the other hand, the training of the new networkshould be managed in such a manner that is does not disturb theperformance that the old network achieved before the addition. Asexplained in the discussions of the other figures, the learning coach 41can learn to optimize the control of the learning of each parameter withcustomized hyperparameters. In FIG. 14, this ability is focused on theissues that are important in this situation by supplying examples of thestudent learning system being trained in similar situations. That is,the learning coach 41 learns to optimize the relative learning rate ofthe new additions to the structure of the student system 11 by observingother examples in which nodes and arcs have been added to a studentsystem 11.

An important special case of the embodiment illustrated in FIG. 14 isthe addition of an entire layer of new nodes and their associated arcsto the network of student learning system 11. In prior art methods, ithas been observed that, as layers are added to a deep neural network,eventually performance degrades not only on test data, but even ontraining data. In the embodiment illustrated in FIG. 15, the learningcoach 41 initializes the weights in the added layer, or othersubstantial new structure, such that the performance of the largernetwork at least matches the performance of the smaller network. Forexample, the learning coach may initialize the weights and biases of theadded structure to be an identity mapping. If necessary, the addedstructure is expanded by adding even more additional nodes with linearor piecewise-linear units so that the added structure can directly matchthe identity function. Then, the learning coach 41 manages the furtherlearning to guarantee that the performance of the now-larger studentlearning system network 11 on training data is always at least as goodon the training data as previous performance of the smaller system. Forexample, whenever a gradient descent step produces a degradation inperformance, the learning coach 41 can decrease the step size. By thedefinition of the gradient, if the gradient is non-zero, there is alwaysa step size such that there is no degradation in performance. Thelearning coach 41 can add nodes to an existing layer in the studentlearning system 11 in the same way. In fact, when adding nodes to anexisting layer, the additional nodes can be initialized to the identityfunction, from the point of view of the rest of the network, simply byinitializing the weights of the output connections from the new nodes tozero. Thus, any addition to an existing network can be made with aguarantee that there will be no degradation in performance.

This embodiment can add additional layers to a deep neural network atany place in the existing student network 11. It can add an additionallayer after the existing output layer, an additional layer just beforethe existing output layer, an additional layer just after the inputlayer, or an additional layer between any two existing layers. It cangrow a neural network to an arbitrary depth without any degradation ofperformance.

The learning coach can also add other substantial structures to thestudent system 11 without degradation in performance. For example, itcan copy a subnetwork of another network, or it could copy a subnetworkfrom one part of student network 11 to another. In other words, it canadd structure (layers and nodes, their connections and their weights) toan existing network, where the added structure replicates the subnetworkbeing copied. In the case of a copied subnetwork, the copying would addadditional features to the location in the network to which thesubnetwork is copied. Thus, the copying would transfer knowledge fromthe source location to the destination location, whether the locationsare in two different networks or in the same network.

Based on the above description, it is clear that embodiments of thepresent invention can be used to improve operation, including thelearning, of machine learning systems, including deep neural networks,in a variety of applications. For example, embodiments of the presentinvention can improve recommender systems, speech recognition systems,and classification systems, including image and diagnosticclassification systems, to name but a few examples, such as by improvingtheir learning hyperparameters, internal structure, and trainingexamples, or any of the other techniques described herein.

The machine learning computer systems described herein can beimplemented with computer hardware and software. For example, they couldbe part of an integrated computer system (e.g., a server or network ofservers) that has multiple processing cores. The program instructionscould be stored in computer memory, such as RAM, ROM, processorregisters or processor cache, etc., that is accessible by the processingcores. The processor cores may comprise CPU or graphical processing unit(GPU) cores. GPU cores operate in parallel and, hence, can typicallyprocess data more efficiently that a collection of CPU cores, but allthe cores execute the same code at one time. As described herein, thevarious machine learning systems could be part of a distributed computersystem (see, e.g., FIGS. 4-5 and 11). Even the embodiments of FIGS. 1,12 and 13 could be implemented as part of a distributed system. In suchembodiments, the computer devices (e.g., servers) that implement thestudents and coaches may be remote from each other and interconnected bydata networks, such as a LAN, WAN, the Internet, etc., using suitablewired and/or wireless data communication links. Data may be sharedbetween the various systems using suitable data links, such as databuses (preferably high-speed data buses) or network links (e.g.,Ethernet).

The software for the various machine learning systems described herein(e.g., the student learning systems 11, 101, etc. and the coach learningsystems 41, 103, 106, etc.) and other computer functions describedherein may be implemented in computer software using any suitablecomputer programming language such as .NET, C, C++, Python, and usingconventional, functional, or object-oriented techniques. For example,the various machine learning systems may be implemented with softwaremodules stored or otherwise maintained in computer readable media, e.g.,RAM, ROM, secondary storage, etc. One or more processing cores (e.g.,CPU or GPU cores) of the machine learning system may then execute thesoftware modules to implement the function of the respective machinelearning system (e.g., student, coach, etc.). Programming languages forcomputer software and other computer-implemented instructions may betranslated into machine language by a compiler or an assembler beforeexecution and/or may be translated directly at run time by aninterpreter. Examples of assembly languages include ARM, MIPS, and x86;examples of high level languages include Ada, BASIC, C, C++, C#, COBOL,Fortran, Java, Lisp, Pascal, Object Pascal, Haskell, ML; and examples ofscripting languages include Bourne script, JavaScript, Python, Ruby,Lua, PHP, and Perl.

In one general aspect, the present invention is therefore directed to amachine learning system that comprises (i) a first student machinelearning system 11 that, using machine learning, automatically learnsfrom and makes predictions on input source data; and (ii) a firstlearning coach machine learning system 41 that is in communication withthe first student machine learning system. Input to the first learningcoach machine learning system comprises data about an internal state ofthe first student machine learning system. Also, the learning coachmachine learning system, using machine learning, automatically learnsand implements an enhancement to the first student machine learningsystem based on the data about the internal state of the first studentmachine learning system to improve operation of the first studentmachine learning system.

In various implementations, the first learning coach machine learningsystem comprises a pattern recognition system that recognizes differentpatterns than the first student machine learning system. Also, the firststudent machine learning system can have a different objective than thefirst student machine learning system. Also, the first learning coachmachine learning system and the first student machine learning systemmay have the same network architectures; for example, either one of themcould comprise a deep neural network.

In various implementation, the enhancement comprises one or more revisedhyperparameters for the first student machine learning system thatimprove learning by the first student machine learning system; astructural change the first student machine learning system; selectivelycontrolling training data input to the first student machine learningsystem to control the learning of the first student machine learningsystem; and/or, where the first student machine learning systemcomprises an ensemble, selectively controlling data flow to members ofthe ensemble or adding new members to the ensemble. The revisedhyperparameters could be a minibatch size for the first student machinelearning system; a learning rate for the first student machine learningsystem; a regularization parameter for the first student machinelearning system; and/or a momentum parameter for the first studentmachine learning system. The structural changes could comprise one ormore additional nodes and/or one or more additional layers to be addedto a selected layer of a network of the first student machine learningsystem.

In addition, the first machine learning system may comprise a computernetwork that comprises: a first computer system that comprises at leastone processor and high-speed memory; and remote secondary storage thatis in communication with the first computer system. In such anembodiment, connection weights and activations for the first studentmachine learning system are stored in the high speed memory so that thefirst student machine learning system can be run when the first studentmachine learning system is active; and the connection weights andactivations for the first student machine learning system are stored inthe secondary storage when the first student machine learning system isnot active.

In other embodiments, either the first student machine learning systemor the first learning coach machine learning system or both comprise agraphics processing unit that comprises multiple processing cores on asingle integrated circuit.

In another general aspect, the present invention is directed to acomputer system that comprises a first set of one or more processingcores; a first set of one or more computer readable media; and a firststudent machine learning module maintained on the first set of one ormore computer readable media that, when executed by the first set of oneor more processing cores, causes the first set of one or more processingcores to, using machine learning, automatically learn from and makepredictions on input source data. The computer system also comprises asecond set of one or more processing cores; a second set of one or morecomputer readable media; and a first learning coach machine learningmodule maintained on the second set of computer readable media. When thefirst learning coach machine learning module is executed by the secondset of one or more processing cores, it causes the second set of one ormore processing cores to: receive as input data about an internal stateof the first student machine learning module; and using machinelearning, automatically learn and implement a change, such as any of thechanges described above, to the first student machine learning modulebased on the data about the internal state of the first student machinelearning module to improve operation of the first student machinelearning module.

In yet another general aspect, the present invention is directed to amethod of improving the operation of a first student machine learningsystem that, using machine learning, automatically learns from and makespredictions on input source data. The method may comprise the steps of:receiving, by a first learning coach machine learning system, from thefirst student machine learning system, data about an internal state ofthe first student machine learning system; and using machine learning,automatically learning and implementing, by the first learning coachmachine learning system, an enhancement to the first student machinelearning system, such as any of the enhancements described above, basedon the data about the internal state of the first student machinelearning system to improve operation of the first student machinelearning system.

The examples presented herein are intended to illustrate potential andspecific implementations of the present invention. It can be appreciatedthat the examples are intended primarily for purposes of illustration ofthe invention for those skilled in the art. No particular aspect oraspects of the examples are necessarily intended to limit the scope ofthe present invention. Further, it is to be understood that the figuresand descriptions of the present invention have been simplified toillustrate elements that are relevant for a clear understanding of thepresent invention, while eliminating, for purposes of clarity, otherelements. While various embodiments have been described herein, itshould be apparent that various modifications, alterations, andadaptations to those embodiments may occur to persons skilled in the artwith attainment of at least some of the advantages. The disclosedembodiments are therefore intended to include all such modifications,alterations, and adaptations without departing from the scope of theembodiments as set forth herein.

What is claimed is:
 1. A machine learning (ML) computer system fortraining a student ML system iteratively through machine learning ontraining data to perform a machine learning task, wherein the student MLsystem comprises: an input layer, an output layer, and at least a firstinner layer between the input and output layers; each layer comprises atleast one node, such that the first inner layer comprises at least afirst node; the first node of the first inner layer outputs anactivation value for each set of input values to the first node of thefirst inner layer; the activation value is determined based on anactivation function for the first node of the first inner layer andbased on learned parameters for the first node of the first inner layer;and following iterations of training of the student ML system, thelearned parameters for the first node of the first inner layer areupdated, and the ML computer system comprises a learning coach ML systemthat is in communication with the student ML system, wherein: thelearning coach ML system comprises a neural network that comprises aninput layer and an output layer; the learning coach ML system has beentrained through machine learning to determine and implement anenhancement to the student ML system based on input to the learningcoach ML system from the student ML system, wherein the input from thestudent ML system is input to the input layer of the learning coach MLsystem; the input to the learning coach ML system from the student MLsystem comprises observations about an internal state of the student MLsystem during training of the student ML system; and the observationsabout the internal state of the student ML system comprise valuesrelated to the learned parameters for the first node on the first innerlayer of the student ML system and the activation values for the firstnode on the first inner layer of the student ML system, such that thelearning coach ML system determines and implements the enhancement tothe student ML system based on the input from the student ML system. 2.The ML computer system of claim 1, wherein the learning coach ML systemcomprises a pattern recognition system that recognizes differentpatterns than the student ML system.
 3. The ML computer system of claim1, wherein the learning coach ML system has a different objective thanthe student ML system.
 4. The ML computer system of claim 1, wherein theenhancement comprises one or more revised hyperparameters for thestudent ML system that improve learning by the student ML system.
 5. TheML system of claim 4, wherein the revised hyperparameter comprises ahyperparameter selected from the group consisting of a mini batch sizefor the student ML system, a learning rate for the student ML system, aregularization parameter for the student ML system and a momentumparameter for the student ML system.
 6. The ML computer system of claim1, wherein the enhancement comprises a structural change to the studentML system.
 7. The ML computer system of claim 6, wherein the structuralchange comprises one or more additional inners layers to be added to thestudent ML system between the input layer and the output layer of thestudent ML system.
 8. The ML computer system of claim 6, wherein thestructural change comprises one or more additional nodes to be added toa selected layer of the one or more inner layers of the student MLsystem.
 9. The ML computer system of claim 8, wherein the learning coachML system implements the one or more additional nodes by providing a setof virtual nodes and activation levels for the virtual nodes associatedwith a particular set of data input values to the student ML system. 10.The ML computer system of claim 9, wherein: the selected layer of thestudent ML system comprises a plurality of existing nodes prior to theimplementation of the enhancement; and after the enhancement: theexisting nodes in the selected layer receive inputs from the virtualnodes; the existing nodes do not back-propagate to virtual nodes duringtraining; the virtual nodes back-propagate to a layer of the student MLsystem below the selected layer; and activations of the virtual nodesare controlled by the learning coach ML system.
 11. The ML computersystem of claim 8, wherein the learning coach ML system implements theone or more additional nodes by providing first and second sets of nodesto be added to the selected layer, wherein: the first and second setsconsist of the same number of nodes; and a learning coach ML systemcontrols a regularization to the second set of nodes so differencesbetween activation values for the second set of nodes and activationvalues for the first set of nodes is less than a threshold value tocontrol a drop-out rate of the nodes in the first and second sets. 12.The ML computer system of claim 8, wherein the learning coach ML systemimplements the one or more additional nodes by providing connectionweights for the one or more additional nodes of the student ML system.13. The ML computer system of claim 1, wherein the enhancement comprisesselectively controlling training data input to the student ML system tocontrol the learning of the student ML system.
 14. The ML computersystem of claim 1, wherein: the ML system comprises a computer networkthat comprises: a first computer system that comprises at least oneprocessor and high-speed memory; and remote secondary storage that is incommunication with the first computer system; connection weights andactivation values for the student ML system are stored in the high speedmemory so that the student ML system can be run when the student MLsystem is active; and the connection weights and activation values forthe student ML system are stored in the secondary storage when thestudent ML system is not active.
 15. The ML system of claim 1, whereinthe learned parameters comprise connection weights and biases for thenodes of the one or more inner layers of the student ML system.
 16. TheML computer system of claim 1, wherein the learning coach ML systemmodels learning performance of the student ML system as a regression.17. The ML computer system of claim 1, wherein the learning coach MLsystem models learning performance of the student ML system as aclassification task.
 18. The ML computer system of claim 1, wherein: thestudent ML system comprises multiple feature nodes, wherein the featurenodes comprise two or more nodes of the student ML system that arefeature nodes for a cluster of data examples within a category; andduring training of the student ML system, activation values for each ofthe multiple feature node are soft-tied for training data exampleswithin the cluster.
 19. The ML computer system of claim 18, whereinsoft-tying the activation values for each of the multiple feature nodesfor training data examples within the cluster comprises using aregularization term in a cost function for each of the multiple featurenodes, wherein the regularization term, for each training data examplein the cluster, for each of the multiple feature nodes, is based on adifference between the activation value of the feature node for thetraining data example and an average activation value across all of themultiple feature nodes for all training data examples in the cluster.20. The ML computer system of claim 19, wherein the regularization term,for each training data example in the cluster, for each of the multiplefeature nodes, is based on a L2 norm of the difference between theactivation value of the feature node for the training data example andthe average activation value across all of the multiple feature nodesfor all training data examples in the cluster.
 21. The ML computersystem of claim 19, wherein the regularization term for each of themultiple feature nodes comprises a relaxation strength hyperparameter.22. The ML computer system of claim 21, wherein the relaxation strengthhyperparameter, for one of the feature nodes, multiplies the differencebetween the activation value of the feature node for the training dataexample and the average activation value across all of the multiplefeature nodes for all training data examples in the cluster.
 23. The MLcomputer system of claim 22, wherein the ML learning coach is configuredto set the relaxation strength hyperparameters for the feature nodes.24. The ML computer system of claim 1, wherein during training of thestudent ML system, an activation value from the first node of thestudent ML system is soft-tied to an activation value of a node in aseparate ML system.
 25. The ML computer system of claim 1, whereinduring training of the student ML system, an activation value from thefirst node of the student ML system is soft-tied to activation values ofmultiple nodes in a separate ML system.
 26. The ML computer system ofclaim 25, wherein soft-tying the activation value from the first node ofthe student ML system to the activation values from the multiple nodesin the separate ML system comprises using a regularization term in acost function for the first node in the student ML system for a firsttraining data example, wherein the regularization term is based on adifference between the activation value for the first node of thestudent ML system for the first training data example and an averageactivation value across all of the multiple nodes in the separate ML towhich the first node is soft tied for the first training data example.27. The ML computer system of claim 26, wherein the regularization termfor the first node comprises a relaxation strength hyperparameter. 28.The ML computer system of claim 27, wherein the relaxation strengthhyperparameter multiplies the difference between the activation value ofthe first node for the first training data example and the averageactivation value across all of the multiple nodes in the separate ML towhich the first node is soft tied for the first training data example.29. The ML computer system of claim 28, wherein the learning coach MLsystem is configured to set the relaxation strength hyperparameter forthe first node.
 30. The ML computer system of claim 1, wherein: thestudent ML system is an ensemble that comprises multiple ML ensemblemembers; and the ML computer system further comprises a data switchingnetwork that selectively directs training data examples for the studentML system to one or more, and less than all, of the multiple ML ensemblemembers.
 31. The ML computer system of claim 30, wherein the dataswitching network selectively directs the training data examples to theone or more of the multiple ML ensemble members based on control signalsfrom the learning coach ML system.
 32. The ML computer system of claim30, further comprising a classifier for classifying the training dataexamples for the student ML system, and wherein the classifier controlsthe data switching network for a training data example based on aclassification by the classifier for the training data example.
 33. Amachine-learning (ML) training method comprising: initially training alearning coach ML system to determine and implement an enhancement to aML system, wherein the learning coach ML system comprises an input layerand an output layer; training, iteratively through machine learning, astudent ML system on training data to perform a machine learning task,wherein: the student ML system comprises an input layer, an outputlayer, and at least a first inner layer between the input and outputlayers; each layer comprises at least one node, such that the firstinner layer comprises at least a first node; the first node of the firstinner layer outputs an activation value for each set of input values tothe first node of the first inner layer; the activation value isdetermined based on an activation function for the first node of thefirst inner layer and based on learned parameters for the first node ofthe first inner layer; and the initial training the student ML systemcomprises, following iterations of the training of the student MLsystem, updating the learned parameters for the first node of the firstinner layer; and training the student ML system comprises: receiving, bythe learning coach ML system, from the student ML system, values relatedto the learned parameters and the activation values of the first node ofthe first inner layer of the student ML system; and determining andimplementing, by the learning coach ML system, the enhancement to thestudent ML system based on the values related to the learned parametersand the activation values of the first node of the inner layer of thestudent ML system to improve performance of the student ML system. 34.The ML training method of claim 33, wherein the enhancement comprisesone or more revised hyperparameters for the student ML system thatimprove learning by the student ML system.
 35. The ML training method ofclaim 33, wherein the enhancement comprises a structural change to thestudent ML system.
 36. The ML training method of claim 35, wherein thestructural change comprises one or more additional inners layers to beadded to the student ML system between the input layer and the outputlayer of the student ML system.
 37. The ML training method of claim 35,wherein the structural change comprises one or more additional nodes tobe added to a selected layer of the one or more inner layers of thestudent ML system.
 38. The ML training method of claim 37, wherein thelearning coach ML system implements the one or more additional nodes byproviding a set of virtual nodes and activation levels for the virtualnodes associated with a particular set of data input values to thestudent ML system.
 39. The ML training method of claim 33, wherein theenhancement comprises selectively controlling training data input to thestudent ML system to control the learning of the student ML system. 40.The ML training method of claim 33, wherein the learned parameterscomprise connection weights and biases for the nodes of the one or moreinner layers of the student ML system.
 41. The ML training method ofclaim 33, wherein: the student ML system comprises multiple featurenodes, wherein the feature nodes comprise two or more nodes of thestudent ML system that are feature nodes for a cluster of data exampleswithin a category; and training the student ML system further comprisessoft-tying activation values for each of the multiple feature nodes fortraining data examples within the cluster.
 42. The ML training method ofclaim 41, wherein soft-tying the activation values for each of themultiple feature nodes for training data examples within the clustercomprises using a regularization term in a cost function for each of themultiple feature nodes, wherein the regularization term, for eachtraining data example in the cluster, for each of the multiple featurenodes, is based on a difference between the activation value of thefeature node for the training data example and an average activationvalue across all of the multiple feature nodes for all training dataexamples in the cluster.
 43. The ML training method of claim 42, whereinthe regularization term, for each training data example in the cluster,for each of the multiple feature nodes, is based on a L2 norm of thedifference between the activation value of the feature node for thetraining data example and the average activation value across all of themultiple feature nodes for all training data examples in the cluster.44. The ML training method of claim 42, wherein the regularization termfor each of the multiple feature nodes comprises a relaxation strengthhyperparameter.
 45. The ML training method of claim 44, wherein therelaxation strength hyperparameter, for one of the feature nodes,multiplies the difference between the activation value of the featurenode for the training data example and the average activation valueacross all of the multiple feature nodes for all training data examplesin the cluster.
 46. The ML training method of claim 45, furthercomprising setting, by the ML learning coach, the relaxation strengthhyperparameters for the feature nodes.
 47. The ML training method ofclaim 33, wherein training the student ML system comprises soft-tying anactivation value from the first node of the student ML system to anactivation value of a node in a separate ML system.
 48. The ML trainingmethod of claim 33, wherein training the student ML system comprisessoft-tying an activation value from the first node of the student MLsystem to activation values of multiple nodes in a separate ML system.49. The ML training method of claim 48, wherein soft-tying theactivation value from the first node of the student ML system to theactivation values from the multiple nodes in the separate ML systemcomprises using a regularization term in a cost function for the firstnode in the student ML system for a first training data example, whereinthe regularization term is based on a difference between the activationvalue for the first node of the student ML system for the first trainingdata example and an average activation value across all of the multiplenodes in the separate ML to which the first node is soft tied for thefirst training data example.
 50. The ML training method of claim 49,wherein the regularization term for the first node comprises arelaxation strength hyperparameter.
 51. The ML training method of claim50, wherein the relaxation strength hyperparameter multiplies thedifference between the activation value of the first node for the firsttraining data example and the average activation value across all of themultiple nodes in the separate ML to which the first node is soft tiedfor the first training data example.
 52. The ML training method of claim51, further comprising setting, by the learning coach ML system, therelaxation strength hyperparameter for the first node.
 53. The MLtraining method of claim 33, wherein: the student ML system is anensemble that comprises multiple ML ensemble members; and the methodfurther comprises selectively directing, by a data switching network,training data examples for the student ML system to one or more, butless than all, of the multiple ML ensemble members.
 54. The ML trainingmethod of claim 53, wherein the data switching network selectivelydirects the training data examples to the one or more of the multiple MLensemble members based on control signals from the learning coach MLsystem.
 55. The ML training method of claim 53, further comprising:classifying, by a classifier, the training data examples for the studentML system; and controlling, by the classifier, the data switchingnetwork for a training data example based on a classification by theclassifier for the training data example.